Repository navigation
fix(chat): recover saved answers and diagnose lost workers - #9120
Conversation
0e3878a to
c68e8e2
Compare
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · Gemini · Google DeepMind · self_reported
Request changes conclusion (author-owned PR; GitHub blocks formal self-review)
[P2] 新增 execution_worker_exited 未接入控制面待处理积压(backlog)判定,导致 health().pending_count 漏计
- 触发条件与路径:在
loopx/chat_runtime.py#L777-L778中,ChatRuntimeController.recover_exited_turn将同进程执行 worker 提前退出(state != LOCK_HOLDER_DEAD)的错误码从原先的runtime_error改为WORKER_EXITED_CODE = "execution_worker_exited"。当该失败通知投递到 Lark 后,LarkPrivateConversations.health()(loopx/extensions/lark/private_conversations.py#L191)通过ChatExternalConversations.quota_backlog_observer()调用 TypeScript 控制面planBoundConversationRequest({pending_observations})(loopx/control_plane/collaboration/conversation_binding.ts#L631-L634)判定原请求是否仍属于未获答复的待处理请求。 - 实测反例证据:在控制面
conversation_binding.ts#L631-L634中,允许保留为pending: true的失败错误码列表仅包含runtime_error与server_restarted等既有错误码(tests/test_private_chat_backlog_projection.py#L101-L107明确验证worker_or_preparation_failure_notice_does_not_answer_the_original_question),并未包含execution_worker_exited。在精确提交c68e8e20b79ded9065adafd82149b0519c605390上通过LarkPrivateConversations.health()与ChatExternalConversations.quota_backlog_pending()构造配对反事实样本实测:runtime_error返回pending: True、server_restarted返回pending: True,而execution_worker_exited返回pending: False,pending_count由应有的3降为2。 - 实际影响:原本作为
runtime_error保留在“待处理或回复”计数中的同进程 worker 退出失败请求,现被静默从pending_count中剔除,影响个人工作台与 Lark 私有连接健康度面板对未答复请求的可见性。 - 最小修复与回归验证:在
loopx/control_plane/collaboration/conversation_binding.ts#L631-L634的失败错误码白名单中补充"execution_worker_exited",并在tests/control_plane_ts/conversation_binding.test.ts与tests/test_private_chat_backlog_projection.py中新增"execution_worker_exited"参数化回归用例,重新运行node --experimental-strip-types --test tests/control_plane_ts/conversation_binding.test.ts与pytest tests/test_private_chat_backlog_projection.py。
动机 / Motivation
受影响的调用方为使用本地 Managed Chat 队列(ChatRuntimeController)以及 Lark 私有对话投递协调路径(LarkPrivateConversations)的操作者与终端用户。
修改前,若响应已保存进入 completing 状态但在发布消息/事件或更新 completed 状态时遇到 I/O 异常,旧 Turn 会持续卡在 completing 阻塞后续 FIFO 排队,或导致原 Lark 回执长期处于未完成;同时非服务端进程死亡的 worker 退出统一退化为 runtime_error,使管家回复无法区分已保存与未保存答复的失败场景。修改后,recover_exited_turn 可在原 execution fence 下幂等完成 completing 态的剩余发布且不重跑模型,Lark 协调路径也能收尾 orphaned completing Turn,并将无保留答复的 worker 退出标记为 execution_worker_exited。
已验证队列与 Lark 原始回执在写入异常注入下仅发布一次原答复、释放后继队列且仅触发一次 turn/start;execution_worker_exited 具有专属诊断文案与脱敏 errno 日志,但该新错误码未同步接入 TypeScript 待处理积压判定导致 pending_count 统计漏计。
本 PR 不重建未保存的答复,不修改部署配置、投递授权范围或执行重试策略。
相比继续放任阻塞(无任何修改)或另建外部重跑队列,在原执行锁(execution fence)与既有 Lark 对账流内做最小幂等收尾能直接消除 FIFO 死锁风险并维持“不重跑模型”的语义。
改动思路 / Approach
本 PR 从“已保存答复的完成态发布中断”与“未保存答复的 worker 退出可观测性”两条路径切入。入口一为 ChatRuntimeController._run_started_turn(loopx/chat_runtime.py#L1127-L1193):当 update_turn(..., status="completing", response=response) 成功后设置 completion_reserved = True,若后续 update_session 或 finalize_managed_turn_completion 发生 OSError,不再误转入 _fail_turn,而是记录含 turn_id、phase=publish_saved_answer 与 exc.errno 的脱敏警告日志后返回,释放文件锁并交由队列或投递协调者收尾。入口二为 ChatRuntimeController.recover_exited_turn(loopx/chat_runtime.py#L730-L793):在获取非破坏性排他文件锁(try_exclusive_file_lock(..., preserve_holder=True))后,若确认 Turn 为托管模式且处于 completing 且无 completion_id(排除 attached host 路径),直接调用 self.store.finalize_managed_turn_completion 幂等补齐助手消息、事件与队列释放;同时 LarkPrivateConversations.reconcile_request(loopx/extensions/lark/private_conversations.py#L1922-L1925)在校验原绑定与来源后,针对 completing 状态补充调用 recover_exited_turn(session_id, turn_id),使即使在 release_active_turn 已执行但 turn.json 未能写成 completed 的窗口内,原始投递回执也能自主推进至终态并发送原答复。
恢复逻辑复用 ChatSessionStore.finalize_managed_turn_completion、execution fence(try_exclusive_file_lock)与 Lark 请求对账边界;但新增状态码 WORKER_EXITED_CODE 跨越 Python 失败写入与 TypeScript 控制面投影(conversation_binding.ts),属于需要同步跨层消费者的契约变更。
保留现有 execution fence 与单轮不自动重放原则,仅在既有 Chat 运行时与 Lark 私有对话协调路径内补全 completing 收尾与 worker 退出诊断;当前需要补齐 TypeScript pending_observations 的错误码对齐后再合并。
具体改动 / Changes
本次精确 head c68e8e20b79ded9065adafd82149b0519c605390 共修改 7 个文件(+164/-24),未涉及成对 RFC 文档变更。其中生产代码涉及 3 个文件(loopx/chat_runtime.py +29/-6、loopx/extensions/lark/manager_context.py +4/-0、loopx/extensions/lark/private_conversations.py +8/-14,含为满足模块行数上限而做的折行压缩),测试代码涉及 4 个文件(+123/-2)。没有现成独立的书面 RFC/Issue 规范,评审依据既有的 chat_store.py 与 chat_runtime.py 契约及现有控制面投影测试。
关键代码讲解
ChatRuntimeController.recover_exited_turn(loopx/chat_runtime.py#L730-L793):在try_exclusive_file_lock(path, preserve_holder=True)确认无存活 worker 持锁并跳过interrupting状态后,优先判定managed and turn and turn.get("status") == "completing" and not turn.get("completion_id")并转交finalize_managed_turn_completion完成幂等发布(即使active_turn_id已被释放也不阻塞)。对于未进入completing的可恢复失败,LOCK_HOLDER_DEAD保留server_restarted,同进程 worker 退出则写入WORKER_EXITED_CODE("execution_worker_exited"),并在complete_recovered_terminalization中保留对历史runtime_error+WORKER_EXITED_MESSAGE组合的兼容收尾。ChatRuntimeController._run_turn与_run_started_turn(loopx/chat_runtime.py#L818-L856,#L929-L1192):_run_turn通过phase变量(start/execute_or_finalize)在捕获OSError时仅输出turn、phase与exc.errno,避免泄露本地路径;_run_started_turn在completing状态落盘后置completion_reserved = True,使发布阶段遭遇OSError时记录phase=publish_saved_answer后平稳退出而非覆盖为失败。manager_failure_reply(loopx/extensions/lark/manager_context.py#L61,#L105-L107):在失败标签表与分支中新增"execution_worker_exited",向用户明确返回“本轮执行任务在完整答复保存前退出,原请求和会话已保留。已有操作可能已生效,系统不会自动重跑这条请求”。LarkPrivateConversations.reconcile_request(loopx/extensions/lark/private_conversations.py#L1922-L1925):在self.bindings.resolve验证授权与来源且turn["status"] == "completing"时,触发recover_exited_turn(record["session_id"], record["turn_id"])并重读 Turn,完成孤立completing态到completed的安全交付;其余改动为行长整理以维持文件在 2000 行上限内。
对主干的风险 / Risks
- 积极与消极路径核验:已核验队列在发布事件遇
EIO后自动从completing恢复并放行下一条排队请求、Lark 回执在release_active_turn后写completed遇EIO时不重跑 provider 而完成唯一投递、以及活锁持有期间调用recover_exited_turn不会越权发布;同时manager_failure_reply不会泄露原始error字段中的私有哨兵文本。 - 跨层语义与积压统计回归(P2 阻塞项):虽然
steer_turn/steward_managed_mode.ts正确地未将execution_worker_exited纳入自动重跑(因为操作可能已生效),且private_web_projection.py#L300-L301能正确复用manager_failure_reply显示新文案,但loopx/control_plane/collaboration/conversation_binding.ts#L631-L634的pending_observations白名单漏掉了"execution_worker_exited",导致与改前"runtime_error"相比发生隐性行为漂移(pending_count漏计)。 - 本地验证摘要(CI 未参考 /
wait_for_ci: false):- 聚焦回归套件(
pytest6 个相关文件共 864 项):PASS(864 passed)。 ruff check(7 个改动文件):PASS。scripts/check_rfc_bilingual.py --base origin/main --head HEAD:PASS。loopx canary premerge --from-git-diff --git-diff-base origin/main(standard 档,2 个 catalog canaries + 8 个 risk-profile smokes):PASS。- 控制面 TS 测试
tests/control_plane_ts/conversation_binding.test.ts(63 passed):PASS(但其枚举尚未覆盖execution_worker_exited)。 - 跨层反事实探测(
quota_backlog_pending/LarkPrivateConversations.health()):FAIL(execution_worker_exited判定为pending=False,runtime_error与server_restarted判定为pending=True)。 mypy --follow-imports=silent(3 个生产文件):PASS(增量无新增失败)(仅存既有方法缺少返回注解的存量告警及chat_runtime.py#L1191裸return)。
- 聚焦回归套件(
- 词汇与语义对齐说明:新增本地错误码常量
WORKER_EXITED_CODE = "execution_worker_exited",目前属于chat_runtime与 Lark/控制面协作层的错误分类扩展,尚未同步注册到conversation_binding.ts的pending_observations集合中。
我的整体评价 / Verdict
本次提交在 completing 态幂等恢复(recover_exited_turn 与 LarkPrivateConversations.reconcile_request)、脱敏 OSError errno 日志和 execution_worker_exited 用户提示方面设计克制且本地故障注入测试充分;从长期运行与用户体验来看,本应显著降低队列卡死与误解风险,但由于未能将新错误码 execution_worker_exited 同步纳入 loopx/control_plane/collaboration/conversation_binding.ts#L631-L634 的待处理积压(backlog)判定集合,导致实际观测到 LarkPrivateConversations.health() 出现 pending_count 静默漏计回归(observable_semantics 存在未预期的行为漂移)。此外需注意 PR 描述中标注的 Tested revision 仍停留在前一提交,而实际审查的精确 head c68e8e20b79ded9065adafd82149b0519c605390 已包含后续行长整理,当前所有本地验证均已在 c68e8e20b7 上完整重跑确认。建议先补充 conversation_binding.ts 与双向回归测试后再复审。无渲染布局改动,visual_review 为不适用(仅核查 manager_context.py 纯文本回复与前端积压引用),未验证范围为无(unverified_scope: none)。
English verdict: REQUEST_CHANGES — exact head c68e8e2 breaks backlog pending-count parity because execution_worker_exited is missing from conversation_binding.ts pending_observations; local suites passed (864 pytest, premerge standard) while the backlog counterfactual failed.
Signed-off-by: LoopX Agent <[email protected]>
Signed-off-by: LoopX Agent <[email protected]>
Signed-off-by: LoopX Agent <[email protected]>
c68e8e2 to
a5d05f0
Compare
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · Gemini · Google DeepMind · self_reported
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
动机 / Motivation
使用本地受管聊天队列(ChatRuntimeController)以及飞书私聊投递协调路径(LarkPrivateConversations)的操作者与个人管家用户,在对话答复已完成生成进入 completing 发布阶段遭遇 I/O 中断,或同进程执行工作线程在保存完整答复前异常退出时触发该问题。
改动前,一旦答复写入 completing 状态后在追加消息、刷新事件或写回 completed 状态时发生文件 I/O 错误,该轮次会长期滞留在 completing 阻塞整条会话 FIFO 队列,或令原始飞书投递回执永远停在未完成状态;同时同进程工作线程提前退出被笼统归为 runtime_error,无法区分已保存答复与未保存答复的退出场景。改动后,recover_exited_turn 与飞书对账入口可在原执行锁下幂等补完已保存答复的唯一发布且不重跑模型,未保存答复的工作线程退出则记为类型化错误码 execution_worker_exited 并同步纳入控制面待处理积压(backlog)统计。
注入发布阶段与终态写盘故障时,队列与飞书私聊回执均仅投递一次已保存答复并正常放行后续请求,模型仅调用一次;无保留答复的同进程退出返回明确的保守不重放提示、脱敏记录阶段与 errno,且与 runtime_error、server_restarted 一致保留在待处理计数(pending_count)中。
本 PR 不重建未保存的模型答复,不自动重放可能已产生外部副作用的未知执行,也不改变部署配置、投递授权范围或上游重试策略。
如果不做这一修复,偶发磁盘满或短暂 I/O 抖动就会把已经算完答案的对话永久卡在队列头部,后续消息全部排队超时;而直接重跑整轮又会破坏一次性副作用边界。在原执行锁与既有对账路径内补齐幂等收尾,并将新增错误码同步接入待处理积压白名单,是解决该故障的最小完整闭环。
改动思路 / Approach
整体思路是在不新增外部重跑器、不改变会话状态机主权的前提下,把“已保存答复的发布收尾”与“未保存答复的退出归因及积压投影”分层收口:
- 已保存答复免覆盖保护与幂等收尾:在
ChatRuntimeController._run_started_turn中,一旦update_turn(..., status="completing", response=response)落盘成功即置completion_reserved = True;此后若update_session或finalize_managed_turn_completion抛出OSError,仅记录脱敏的turn、phase=publish_saved_answer与exc.errno并退出,绝不调用_fail_turn冲掉已保存答复。随后由队列推进或LarkPrivateConversations.reconcile_request调用ChatRuntimeController.recover_exited_turn,在try_exclusive_file_lock(..., preserve_holder=True)确认原执行线程已退出且非attached模式(无completion_id)后,复用store.finalize_managed_turn_completion完成唯一消息写入、终态事件发布与活跃轮次释放。 - 工作线程退出类型化归因与控制面积压对齐:对于未进入
completing的异常退出,recover_exited_turn区分跨进程死锁持有者(保留server_restarted)与同进程工作线程丢失(写入WORKER_EXITED_CODE = "execution_worker_exited"),同时在complete_recovered_terminalization兼容旧版runtime_error退出记录。本次精确提交进而在 TypeScript 控制面planBoundConversationRequest的pending_observations失败码白名单中补入"execution_worker_exited",消除了上一提交c68e8e20b7中LarkPrivateConversations.health()漏计待处理积压的回归。 - 架构比例与边界判断:状态不变量留守 ChatSessionStore 与每轮文件锁,积压判定策略留守 TypeScript 控制面 conversation_binding.ts,飞书扩展仅作薄调用与文案投影,无需拆分或新增组件。当前 PR 边界严格限定于已保存答复的幂等发布收尾、同进程工作线程退出诊断与积压统计一致性;预分发阶段自动接续由独立下游切片 #9125 承接。
具体改动 / Changes
精确审查提交为 a5d05f0284d795307cf86c89b522625967847d6c(基线 origin/main 5eb6b24143bc4d421f665e95527bb332a9476313),共涉及 10 个文件(+167/-27):4 个生产文件(+42/-21)与 6 个测试文件(+125/-6),无成对双语 RFC 文件变更。仓库内无专门针对 completing 发布中断收尾与 execution_worker_exited 错误码的独立成文 RFC 或协议文档,本次依据既有会话状态机与控制面积压不变量审查。评审对照 loopx/chat_store.py、loopx/chat_runtime.py 与 loopx/control_plane/collaboration/conversation_binding.ts 的既有完成围栏、单次投递与积压投影不变量逐项核验。
关键代码讲解
ChatRuntimeController.recover_exited_turn:在非破坏性排他锁下,若受管轮次处于completing且无completion_id,直接委托finalize_managed_turn_completion幂等补完发布(即便active_turn_id已被释放仍可由飞书对账路径收尾);若未达completing,按锁状态写入server_restarted或execution_worker_exited,并保留interrupting(用户/stop)优先不被改写。ChatRuntimeController._run_turn与_run_started_turn:细化phase标记并在completion_reserved成立后拦截发布期OSError,仅记录errno而不泄露本机路径或异常原文。LarkPrivateConversations.reconcile_request与manager_failure_reply:前者在通过绑定与来源核验后对孤立completing轮次触发recover_exited_turn收口(并将文件保持在 1996 行预算内);后者为execution_worker_exited提供明确的中文保守提示。planBoundConversationRequest:将"execution_worker_exited"纳入未决失败观察码列表,确保工作线程异常退出后未获实质答复的私聊请求继续计入pending_count。
对主干的风险 / Risks
- 正向、负向与反事实核验:
- PASS(已保存答复幂等恢复与队列解封):在事件写出与终态更新阶段分别注入
EIO时,原轮次与后续排队轮次均恰好完成一次,飞书回执仅收到一条终态回复,上游模型仅启动一次。 - PASS(活锁保护与并发
/stop边界):当工作线程仍持有活跃执行锁或轮次处于interrupting/attached(含completion_id)时,recover_exited_turn严格拒绝越权收尾或覆盖取消状态。 - PASS(上一轮
[P2]积压漏计回归已闭环):在a5d05f0284d795307cf86c89b522625967847d6c上运行配对反事实探针(fixture_history(count=3)分别注入runtime_error、execution_worker_exited、server_restarted),LarkPrivateConversations.health()返回pending_count: 3, recovery_count: 0(修复前c68e8e20b7为2),且底层持久化记录零副作用。
- PASS(已保存答复幂等恢复与队列解封):在事件写出与终态更新阶段分别注入
- 本地验证矩阵(遵循 Goal 配置
wait_for_ci=false,未查询或等待远端 CI):- PASS:
node --experimental-strip-types --test tests/control_plane_ts/conversation_binding.test.ts(65 passed, 0 failed)与tsc --project tsconfig.control-plane.json --noEmit(0 errors)。 - PASS:聚焦 Python 回归套件
pytest tests/test_private_chat_backlog_projection.py tests/test_chat_queue_preparation.py tests/test_chat_runtime_activation.py tests/test_lark_private_conversations.py tests/test_steward_interrupted_answer.py(847 passed)。 - PASS:
ruff check(8 个改动 Python 文件全过)、scripts/check_rfc_bilingual.py --base origin/main --head HEAD、examples/semantic-vocabulary-drift-smoke.py、loopx check --scan-path ...(ok: true,errors: [])及loopx canary premerge --tier quick --from-git-diff --git-diff-base origin/main(4 项直接检查 + 2 项 catalog canaries 含 maintainability ratchet 与 semantic drift 全绿,0 失败)。
- PASS:
- 残余边界说明:测试在真实本地存储、文件锁与控制面进程上通过故障注入完成;未对线上运行中的守护进程直接注入故障或重启。
我的整体评价 / Verdict
精确提交 a5d05f0284d795307cf86c89b522625967847d6c 完整解决了已保存答复在发布期遇 I/O 异常卡死队列与飞书回执的缺陷,并彻底修复了上一版 c68e8e20b7 遗留的 execution_worker_exited 待处理积压漏计问题(problem_context 结论为 goal_achieved)。从结果影响看,long_horizon 为 improved(消除了无需重跑模型即可恢复的队列死锁且严守不重放未知副作用底线),user_experience 为 improved(恢复已生成答复投递、区分线程退出诊断并保持健康度积压计数准确)。视觉审查(visual_review)判定为 not_applicable:PR #9120 仅修改后端会话运行时(loopx/chat_runtime.py)、飞书私聊协调与纯文本失败提示(loopx/extensions/lark/private_conversations.py、loopx/extensions/lark/manager_context.py)以及控制面积压错误码列表(loopx/control_plane/collaboration/conversation_binding.ts),无前端页面布局或交互组件变更。 未验证范围(unverified_scope):none。代码体量克制、职责归属清晰,满足合入条件。
English verdict: APPROVE at a5d05f0. Saved-answer publication recovery, sanitized worker-exit diagnostics, and pending-backlog accounting parity are verified across 847 Python tests, 65 TypeScript tests, paired counterfactual probes, and quick premerge checks.
* fix(cli): preserve typed runtime diagnostics in Turn failures Signed-off-by: LoopX Agent <[email protected]> * feat(steward): support per-endpoint model preferences and Codex-to-Antigravity failover (#9141) Signed-off-by: LoopX Agent <[email protected]> * Keep the prepared Chat adapter available to a retry (#9134) A failed worker start can finish cleanup after a same-turn retry has selected the adapter. Keep its lifetime tied to the Session and let failed dispatch release its own worker registration. The regression asserts one upstream turn and explicit adapter closure with the Session. Port the fix from 4453e4bdd77486ec14b952f88ca893310b32e26c. (cherry picked from commit 173e42b4f051b885c3a3389e1cad9c5722f7b7dd) Signed-off-by: LoopX Agent <[email protected]> * fix(steward): recover zero-effect monitors and cleared-replan reports (#9143) Signed-off-by: LoopX Agent <[email protected]> * fix(steward): admit flexible_primary_available executor transitions in conversation_scope (#9145) Signed-off-by: LoopX Agent <[email protected]> * Verify quota gate scope through signed lane references (#9101) * test(quota): verify scoped gates through complete lane references Signed-off-by: LoopX Agent <[email protected]> * Test canonical quota gates in both public response modes Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * Test C1 planning and Host input from the original quota capture (#9102) * Use recorded quota decisions in the no-Todo settlement regression Signed-off-by: LoopX Agent <[email protected]> * Exercise the complete Host prompt from the captured quota source Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * Prepare the CLI budget baseline with a private Git index (#9103) Signed-off-by: LoopX Agent <[email protected]> * Exercise C1 Goal delivery and bound replan from the saved source (#9104) Signed-off-by: LoopX Agent <[email protected]> * Keep long claim overlap scopes on demand (#9086) * Link long thin-claim overlap lists to their receipts Signed-off-by: LoopX Agent <[email protected]> * Read claim overlap details without copying the lease Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(quota): preserve explicit Monitor Todo selection (#9099) Carry Todo selection provenance into quota admission. Preserve explicit secondary Monitor polling and recovery from previously admitted receipts. Signed-off-by: LoopX Agent <[email protected]> * Keep ready Todo scope details on the required context read (#9106) Signed-off-by: LoopX Agent <[email protected]> * Fix archive rollback with custom install directories (#9066) * refactor(core): defer benchmark implementations during capability discovery Preserve the benchmark toolkit public exports and static types while keeping execution and experiment implementations out of metadata discovery. Qualify source and installed Core paths, and reconcile the paired minimal/Standard RFC boundary without changing activation or authority defaults. Signed-off-by: cicolord <[email protected]> * Defer Yardstick evaluation until an enabled Explore read Signed-off-by: cicolord <[email protected]> * Omit empty Todo summary lanes from ordinary Worker context Signed-off-by: cicolord <[email protected]> * fix(update): honor configured archive directories during rollback Signed-off-by: LoopX Agent <[email protected]> * fix(lark): replay recall guard on current main Signed-off-by: huangruiteng <[email protected]> * docs: record the verified LoopX 2.3.0 release baseline Signed-off-by: LoopX Agent <[email protected]> * Preserve quota spend source when prepared work is admitted Signed-off-by: LoopX Agent <[email protected]> * test(quota): qualify native saved complete-Turn checkpoints Signed-off-by: LoopX Agent <[email protected]> * Retry cleanup of an owned file mutation marker Signed-off-by: LoopX Agent <[email protected]> * Keep synthetic Git-path test data outside the public source scan Signed-off-by: LoopX Agent <[email protected]> * fix(quota): share native preparation wait with readiness Signed-off-by: LoopX Agent <[email protected]> * fix(doctor): bind package update readback to its Python environment Signed-off-by: LoopX Agent <[email protected]> * Keep Doctor discovery separate from service activation Signed-off-by: LoopX Agent <[email protected]> * Verify the recorded Python launcher before Doctor probes it Signed-off-by: LoopX Agent <[email protected]> * fix(desktop): require runtime qualification for macOS preview updates Signed-off-by: LoopX Agent <[email protected]> * fix(quota): reuse replan projection owners during recovery Signed-off-by: LoopX Agent <[email protected]> * Qualify Codex 0.169 native context replay on main Signed-off-by: LoopX Agent <[email protected]> * Stop regenerating retired quota view aliases Signed-off-by: LoopX Agent <[email protected]> * fix: pin deferred native context reads to the original capture Signed-off-by: LoopX Agent <[email protected]> * Keep native capture navigation within small output budgets Signed-off-by: LoopX Agent <[email protected]> * Retire legacy heartbeat work prompt generation Signed-off-by: LoopX Agent <[email protected]> * Capture C1 composition failures as receipt diagnostics Signed-off-by: LoopX Agent <[email protected]> * Recover interrupted native Todo readiness from the original Turn Signed-off-by: LoopX Agent <[email protected]> * Revalidate the original lease before publishing resumed native context Signed-off-by: LoopX Agent <[email protected]> * Fence native readiness publication with the canonical lease writer Signed-off-by: LoopX Agent <[email protected]> * fix(scheduler): keep Codex fallback on the original C1 runtime Signed-off-by: LoopX Agent <[email protected]> * fix(scheduler): pin the originating fallback launchers Signed-off-by: LoopX Agent <[email protected]> * Retire the duplicate Codex App scheduler projection Signed-off-by: LoopX Agent <[email protected]> * Seed the recorded scheduler profile in the elapsed-backoff fixture Signed-off-by: LoopX Agent <[email protected]> * Verify canonical cadence on refused and settled Turns Signed-off-by: LoopX Agent <[email protected]> * Pin named Core commands in saved scheduler fallbacks Signed-off-by: LoopX Agent <[email protected]> * fix(research): preserve saved case addresses when a version cannot open Signed-off-by: LoopX Agent <[email protected]> * Preserve citation focus after a saved study refresh Signed-off-by: LoopX Agent <[email protected]> * Capture authenticated LoopX receipts for Research attention (#8790) * feat(research): capture authenticated LoopX attention receipts Signed-off-by: lusendong.6789 <[email protected]> Co-authored-by: TRAE CLI <[email protected]> * fix(research): tighten attention receipt provenance Signed-off-by: lusendong.6789 <[email protected]> Co-authored-by: TRAE CLI <[email protected]> * fix(research): publish attention captures after source validation Signed-off-by: huangruiteng <[email protected]> --------- Signed-off-by: lusendong.6789 <[email protected]> Signed-off-by: huangruiteng <[email protected]> Co-authored-by: TRAE CLI <[email protected]> Co-authored-by: huangruiteng <[email protected]> * Recover the original Session and Turn from a confirmed Chat task Signed-off-by: huangruiteng <[email protected]> * perf(lark,native_chat): eliminate redundant full-directory JSON reparsing in steward chat loops (#9041) * perf(lark,native_chat): avoid full directory reparsing in steward chat loops - Make LarkOwnerAttention._route iterate capture candidates lazily so matching stops at the newest valid p2p capture. - Reuse FileChatDiscovery in PrivateIngressObserver._records while preserving strict field validation and fail-closed errors. - Reuse FileChatDiscovery in ChatExternalConversations.pending so unchanged request journals skip redundant JSON reads and stay mutation-isolated. - Add regression and scan-efficiency tests in tests/test_lark_steward_directory_scan_perf.py. Signed-off-by: LoopX Agent <[email protected]> * fix(native-chat): project small recovery metadata fields to stop data_url memory growth Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * Retire transitional smoke checks and duplicate execution (#9073) Signed-off-by: lusendong.6789 <[email protected]> Co-authored-by: TRAE CLI <[email protected]> * feat(steward): support probe-gated generic continuous_monitor todos with handoff and user actions (#9074) Signed-off-by: LoopX Agent <[email protected]> * Remove redundant defensive prose from prompts and comments (#9072) Remove redundant defensive prose while preserving published heartbeat wrappers and externally managed skill readiness. Validated with 197 related tests, independent historical wrapper migrations and negative cases, installed-skill readback, and current-main quick canary. Existing CI qualification and infrastructure limitations remain recorded in PR #9072. Signed-off-by: lusendong.6789 <[email protected]> Signed-off-by: Lusen Dong <[email protected]> * Keep original Chat retry errors visible during receipt refresh Signed-off-by: LoopX Agent <[email protected]> * fix(steward): tolerate report validator python drift and allow cancel of mid-turn failed reports (#9077) Signed-off-by: LoopX Agent <[email protected]> * feat(steward): add exact-head PR review signals and protected merge confirmation (#9081) Signed-off-by: LoopX Agent <[email protected]> * feat(steward): add isolated OpenViking long-term memory with per-speaker boundaries (#9083) Signed-off-by: LoopX Agent <[email protected]> * refactor(content-ops): slim surface module and eliminate Any (#8961) * refactor(content-ops): slim surface module and eliminate Any Signed-off-by: LoopX Agent <[email protected]> * fix(content-ops): reject malformed angle source_item_ids containers Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(collaboration): batch handoff query Goal lifetime guard lock per Goal (#9085) Signed-off-by: LoopX Agent <[email protected]> * fix(hosts): raise managed service startup readiness timeout to 45s (#9087) Signed-off-by: LoopX Agent <[email protected]> * perf(manager-context): memoize history_inspect lifecycle checks and scope runtime revision in handoffs query (#9088) Eliminate the remaining O(N) Effect runtime RPCs and uncached runtime fingerprinting inside tracking.query(): enter effect_runtime_request_scope() for the query, memoize pure history_inspect lifecycle decisions across entries sharing identical Goal refs while holding the Goal lifetime guard, lazily discover legacy ingress files only when external audience recovery needs them, and preserve entry scan order across Goal batches. Signed-off-by: LoopX Agent <[email protected]> * refactor(lark): slim private_conversations below 2000 lines with net LOC reduction (#8929) Signed-off-by: LoopX Agent <[email protected]> * perf(chat,hosts): skip frontend source re-hashing on immutable installed releases and extend readiness timeout to 90s (#9089) Signed-off-by: LoopX Agent <[email protected]> * refactor(control-plane): split and slim quota monitor_poll_commit (#8885) * refactor(control-plane): split and slim quota monitor_poll_commit Signed-off-by: LoopX Agent <[email protected]> * fix(control-plane): preserve legacy lane budget and observation diagnostic order Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * refactor(quota): slim loopx/quota.py and tighten Any count to zero (#8939) Signed-off-by: LoopX Agent <[email protected]> * refactor(native-chat): slim external_conversations below 2k lines and retire ceiling (#8911) * refactor(native-chat): move steward commission mixin to steward_commands and retire external_conversations ceiling Signed-off-by: LoopX Agent <[email protected]> * fix(native-chat): respect include_raw_response in retained result recovery Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * refactor(chat): slim chat_runtime below 1,500 lines, fix entry-point project_contexts guard, and retire ceiling (#8920) * refactor(chat): move session lifecycle mixin to session_open, deduplicate runtime helpers, and retire ceiling Signed-off-by: LoopX Agent <[email protected]> * fix(chat): drop turn_done_events slot before waking waiters on Thread.start failure Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(steward): paginate open PR signal GraphQL, allow signal-unavailable monitor recovery, and cache private delivery scans (#9093) Signed-off-by: LoopX Agent <[email protected]> * perf(lark,native_chat): back off noop recovery retries and cache recovery lookups (#9096) Signed-off-by: LoopX Agent <[email protected]> * perf(native_chat,lark): cache unchanged quota backlog health and interrupt observations (#9098) Signed-off-by: LoopX Agent <[email protected]> * perf(lark,native_chat): coalesce listener snapshots and cache recovery event summaries (#9100) Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: cicolord <[email protected]> Signed-off-by: LoopX Agent <[email protected]> Signed-off-by: huangruiteng <[email protected]> Signed-off-by: lusendong.6789 <[email protected]> Signed-off-by: Lusen Dong <[email protected]> Co-authored-by: cicolord <[email protected]> Co-authored-by: huangruiteng <[email protected]> Co-authored-by: Lusen Dong <[email protected]> Co-authored-by: TRAE CLI <[email protected]> * refactor(issue-fix): consolidate maintainer-correction admission and slim pr_lifecycle (#8897) Signed-off-by: LoopX Agent <[email protected]> * Fix truncated requests and responses in the Codex API relay (#8713) * fix(codex relay): fence incomplete model requests before drain Signed-off-by: LoopX Agent <[email protected]> * fix(codex relay): surface incomplete upstream responses Port the verified response framing fix to the request-admission branch. Original candidate 27bc3aee311cdfc6a00466ff9d3b58c7db36e01b. Signed-off-by: LoopX Agent <[email protected]> * Bound Codex relay writes to clients that stop reading Signed-off-by: LoopX Agent <[email protected]> * Reject upstream chunked responses missing the trailer terminator Signed-off-by: LoopX Agent <[email protected]> * Validate upstream chunk data delimiters before relay completion Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(release): repair dsh closeout test, antigravity failover alias, and release download retry (#9147) Signed-off-by: LoopX Agent <[email protected]> * docs(release): update v2.4.0 release notes and dsh-loopx-plugin 0.1.1-beta.7 README (#9149) Signed-off-by: LoopX Agent <[email protected]> * fix(chat): recover saved answers and diagnose lost workers (#9120) * fix(chat): recover saved answers and diagnose lost workers Signed-off-by: LoopX Agent <[email protected]> * style(lark): keep private_conversations within module line budget Signed-off-by: LoopX Agent <[email protected]> * fix(chat): retain execution_worker_exited in pending backlog count Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(chat): resume steward requests lost before dispatch (#9125) Signed-off-by: LoopX Agent <[email protected]> * fix(benchmark): restore frozen v1 archives and align checkpoint/runtime tests with C1 defaults (#9151) Signed-off-by: LoopX Agent <[email protected]> * fix(release): stabilize CLI help, architecture guards, onboarding gate, and public smokes (#9152) Signed-off-by: LoopX Agent <[email protected]> * fix(steward): recover report selection across python upgrades and prevent queue kill on observation monitors (#9155) Signed-off-by: LoopX Agent <[email protected]> * fix(release): align extensions, chat, presentation, and transport tests for v2.4.0 (#9156) Signed-off-by: LoopX Agent <[email protected]> * fix(control-plane): stabilize v2.4.0 quota, turn-envelope, and capabilities slice (Slice B1) (#9154) * fix(control-plane): stabilize v2.4.0 quota, turn-envelope, and capabilities slice Signed-off-by: LoopX Agent <[email protected]> * fix(quota): preserve tokenless identity and exact numeric types in attach_complete_turn_context Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(steward): recover frozen report contract failures and rebind resumed task leases (#9157) Signed-off-by: LoopX Agent <[email protected]> * fix(steward): recover pre-dispatch runtime failures across endpoints and stop receipt-unknown disk churn (#9159) Signed-off-by: LoopX Agent <[email protected]> * fix(control_plane): align lease recovery, settlement readback, and runtime fixtures (#9158) * fix(control_plane): align lease recovery, settlement readback, and runtime fixtures Signed-off-by: LoopX Agent <[email protected]> * fix(control_plane): forward record_capture in worker outage test and harden version type guard Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(control-plane): stabilize v2.4.0 explore, context, and checkpoint tail slice (Slice B4) (#9161) Signed-off-by: LoopX Agent <[email protected]> * fix(control-plane): stabilize v2.4.0 replan, monitor recovery, and staged-read slice (Slice B5) (#9160) Signed-off-by: LoopX Agent <[email protected]> * docs(release): reconcile final v2.4.0 stabilization PRs and boundary wording (#9162) * docs(release): reconcile final v2.4.0 stabilization PRs and boundary wording Signed-off-by: LoopX Agent <[email protected]> * docs(release): align #9158 and #9161 descriptions with verified commits Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> * fix(steward): tolerate minor JSON delimiter glitches and recover saved report candidates (#9164) Signed-off-by: LoopX Agent <[email protected]> * fix(control-plane): preserve bound yardstick reversals in native replan openings (#9165) Signed-off-by: LoopX Agent <[email protected]> * fix(native-chat): recover loopback relay 401 failures and isolate test auth (#9167) Signed-off-by: LoopX Agent <[email protected]> --------- Signed-off-by: LoopX Agent <[email protected]> Signed-off-by: cicolord <[email protected]> Signed-off-by: huangruiteng <[email protected]> Signed-off-by: lusendong.6789 <[email protected]> Signed-off-by: Lusen Dong <[email protected]> Co-authored-by: cicolord <[email protected]> Co-authored-by: huangruiteng <[email protected]> Co-authored-by: Lusen Dong <[email protected]> Co-authored-by: TRAE CLI <[email protected]>
Goal And Delivered Outcome
execution_worker_exitedresult. The notice describes what is known and says the request is retained and will not be replayed. Captured checkpoint I/O failures log the stage and errno without the raw exception text.loopx-project/loopx:main.Author Declaration
loopx/chat_store.pyandloopx/chat_runtime.pyatcc5392e9538e81487e9e551b915dabc68c085c25.ChatRuntimeController.recover_exited_turnLarkPrivateConversations.reconcile_requestScope And Continuation
This repair covers publication of a reserved Chat answer and diagnosis of a lost execution worker. It doesn't reconstruct an unsaved answer or change deployment, delivery grants, or execution retry policy.
Validation
a5d05f0284d795307cf86c89b522625967847d6c.regression_parityintegrationintegrationstaticgit diff --check, RFC pair check, and a public boundary scan on changed paths.Provider adapters run with synthetic inputs. The live service was not restarted or used for fault injection.
Frontend / Visual Evidence
No layout or interaction change. The only visible copy change is the plain-text terminal notice for the specific lost-worker case, tested through the existing reply projection.
Type of Change
LoopX Area
Shared-authority RFC fixture impact
N/A; existing authority and persistence contracts remain in use.
Boundary Checklist